fix: optimize sensitive log parsing performance and reduce false positives by mayankpande88 · Pull Request #17 · nudgebee/logparser

mayankpande88 · 2026-03-13T09:20:18Z

Summary

Add anchor-based pre-filtering that skips ~91% of regex evaluations on normal log lines (21-55x faster)
Add confidence tiers (high/medium/low) to all 200 sensitive patterns, with configurable minimum confidence level
Add Shannon entropy + character-class validation to reject false positives on low-confidence patterns (e.g. SQL table names, cache keys)
Add singleton pattern cache to share compiled regexes across parsers (~14KB vs ~2.1MB per parser)
Replace bare bool parameter with SensitiveConfig struct supporting sampling rate, min confidence, and max detections cap
Add configurable 1-in-N line sampling and per-container detection cap

Test plan

go test ./... passes
TestExtractAnchors validates anchor extraction from regex patterns
TestPrefilterEffectiveness confirms ≥80% anchor coverage
TestPrefilterSkipsNonMatching confirms ≥90% skip rate on normal lines
TestLooksLikeSecret validates entropy-based FP rejection
TestDetectSensitiveData_LowConfidence_EntropyFilter confirms real secrets detected, SQL table names rejected
TestParserSensitiveDisabled confirms zero overhead when disabled
TestParserSensitiveSampling and TestParserSensitiveMaxDetections validate rate limiting
Benchmark: go test -bench=. -benchmem shows improvement over baseline

…ence tiers, and sampling - Add anchor-based pre-filtering to skip ~91% of regex evaluations on normal log lines - Add confidence tiers (high/medium/low) to sensitive_patterns.json for all 200 patterns - Add Shannon entropy + character-class validation to reject false positives on low-confidence patterns - Add singleton pattern cache to share compiled regexes across all parsers (~14KB vs ~2.1MB per parser) - Replace bare bool parameter with SensitiveConfig struct (sampling rate, min confidence, max detections) - Add configurable sampling (1-in-N lines) and per-container detection cap

gemini-code-assist · 2026-03-13T09:20:41Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a comprehensive set of enhancements to the sensitive log parsing mechanism. The primary goal is to drastically improve performance and accuracy by intelligently filtering log lines before full regex evaluation and by applying more robust validation to potential matches. These changes provide greater control over the detection process and ensure that the system is both efficient and reliable in identifying sensitive data.

Highlights

Performance Optimization: Implemented anchor-based pre-filtering that significantly reduces regex evaluations on normal log lines, leading to 21-55x faster processing.
False Positive Reduction: Introduced Shannon entropy and character-class validation to reject false positives, especially for low-confidence patterns like SQL table names or cache keys.
Configurable Confidence Tiers: Assigned confidence tiers (high, medium, low) to all 200 sensitive patterns, allowing for configurable minimum confidence levels.
Memory Efficiency: Added a singleton pattern cache to share compiled regexes across parsers, reducing memory footprint from ~2.1MB to ~14KB per parser.
Enhanced Configuration: Replaced a bare boolean parameter with a SensitiveConfig struct, enabling granular control over sampling rate, minimum confidence, and maximum detections per container.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

bench_loki_test.go
- Added a new benchmark file to test sensitive data detection performance against real logs fetched from Loki.
bench_test.go
- Added new benchmarks to measure the performance of sensitive data detection, including comparisons with the old approach and pre-filter-only scenarios.
cmd/logparser.go
- Updated the NewParser instantiation to utilize the new SensitiveConfig struct for sensitive data detection.
impact_test.go
- Added a new test file to measure the CPU and memory resource impact of sensitive data detection under various configurations.
parser.go
- Added strings package import.
- Introduced a global patternCache with mutex for sharing compiled regexes across parsers.
- Modified PrecompiledPattern struct to include Anchors for pre-filtering and Confidence level.
- Defined SensitiveConfig struct to encapsulate sensitive data detection parameters.
- Updated Parser struct to store SensitiveConfig and a sensitiveCounter for sampling.
- Refactored NewParser to accept SensitiveConfig and load patterns using the new caching mechanism.
- Converted processSensitivePattern to a method of Parser and integrated sampling and max detections logic.
- Updated SensitivePattern struct to include a Confidence field for JSON unmarshalling.
- Added confidenceLevel helper function to map confidence strings to numeric levels.
- Modified DetectSensitiveData to incorporate anchor-based pre-filtering and looksLikeSecret post-validation for low-confidence patterns.
- Implemented getOrLoadPatterns to retrieve patterns from the shared cache or load them if not present.
- Updated LoadPatterns to filter patterns based on minConfidence and to extract anchors for each pattern.
parser_test.go
- Updated TestParser to initialize the parser with SensitiveConfig.
- Added TestParserSensitiveDisabled to verify that sensitive detection has no overhead when disabled.
- Added TestParserSensitiveSampling to test the 1-in-N line sampling functionality.
- Added TestParserSensitiveMaxDetections to ensure the detection cap works correctly.
- Added TestParserMinConfidence to validate the filtering of patterns by confidence level.
- Initialized sensitivePatterns in TestParserCardinalityLimit for consistency.
sensitive_filter.go
- Added a new file containing utility functions for sensitive data filtering.
- Implemented extractAnchors to derive literal substrings from regexes for pre-filtering.
- Added leadingLiteral to extract literal prefixes from strings.
- Created anchorMatchesLine to efficiently check for anchor presence in a log line.
- Developed shannonEntropy to calculate the entropy of a string.
- Implemented looksLikeSecret to validate potential sensitive matches using entropy and character-class analysis.
- Added dedupStrings helper function.
sensitive_filter_test.go
- Added a new test file for the sensitive filtering logic.
- Included TestExtractAnchors to validate anchor extraction from various regex patterns.
- Added TestAnchorMatchesLine to test the anchor matching functionality.
- Implemented TestPrefilterEffectiveness to verify the coverage of anchor extraction across patterns.
- Added TestPrefilterSkipsNonMatching to confirm the pre-filter's ability to skip irrelevant lines.
- Included TestLooksLikeSecret to test the false positive rejection logic based on entropy and character classes.
- Added TestShannonEntropy to verify entropy calculation.
- Implemented TestDetectSensitiveData_LowConfidence_EntropyFilter to confirm correct detection and false positive rejection for low-confidence patterns.
- Added TestDetectSensitiveDataWithPrefilter to test sensitive data detection with pre-filtering enabled.

Activity

The author mayankpande88 initiated this pull request to address performance and accuracy issues in sensitive log parsing.
The pull request includes a detailed summary of the changes and a comprehensive test plan, indicating thorough consideration of the new features.
New benchmark and impact tests have been added to validate the performance improvements and resource usage of the updated detection mechanisms.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces significant performance optimizations and accuracy improvements for sensitive data detection in logs. The changes include anchor-based pre-filtering, confidence levels for patterns, entropy-based validation, and a shared cache for compiled regexes. The implementation is solid and backed by extensive new benchmarks and tests. I've found a few areas for improvement, mainly related to test robustness and minor code cleanup, which I've detailed in the comments.

gemini-code-assist Bot reviewed Mar 13, 2026

View reviewed changes

Comment thread bench_loki_test.go

Comment thread bench_loki_test.go

Comment thread bench_loki_test.go

Comment thread impact_test.go

Comment thread impact_test.go

Comment thread parser.go Outdated

Comment thread sensitive_filter.go Outdated

RamanKharchee previously approved these changes Mar 13, 2026

View reviewed changes

fix: remove dead code and make prefix stripping robust

a30d466

mayankpande88 dismissed RamanKharchee’s stale review via a30d466 March 13, 2026 09:56

RamanKharchee approved these changes Mar 13, 2026

View reviewed changes

mayankpande88 merged commit b22168b into main Mar 13, 2026

mayankpande88 mentioned this pull request Mar 13, 2026

fix: configurable sensitive log detection with optimized logparser nudgebee/node-agent#224

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: optimize sensitive log parsing performance and reduce false positives#17

fix: optimize sensitive log parsing performance and reduce false positives#17
mayankpande88 merged 2 commits into
mainfrom
fix-sensitive-log-parsing

mayankpande88 commented Mar 13, 2026

Uh oh!

gemini-code-assist Bot commented Mar 13, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

mayankpande88 commented Mar 13, 2026

Summary

Test plan

Uh oh!

gemini-code-assist Bot commented Mar 13, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants